Dynamic Vision Sensor integration on FPGA-based CNN accelerators for high-speed visual classification

Deep-learning is a cutting edge theory that is being applied to many fields. For vision applications the Convolutional Neural Networks (CNN) are demanding significant accuracy for classification tasks. Numerous hardware accelerators have populated during the last years to improve CPU or GPU based solutions. This technology is commonly prototyped and tested over FPGAs before being considered for ASIC fabrication for mass production. The use of commercial typical cameras (30fps) limits the capabilities of these systems for high speed applications. The use of dynamic vision sensors (DVS) that emulate the behavior of a biological retina is taking an incremental importance to improve this applications due to its nature, where the information is represented by a continuous stream of spikes and the frames to be processed by the CNN are constructed collecting a fixed number of these spikes (called events). The faster an object is, the more events are produced by DVS, so the higher is the equivalent frame rate. Therefore, these DVS utilization allows to compute a frame at the maximum speed a CNN accelerator can offer. In this paper we present a VHDL/HLS description of a pipelined design for FPGA able to collect events from an Address-Event-Representation (AER) DVS retina to obtain a normalized histogram to be used by a particular CNN accelerator, called NullHop. VHDL is used to describe the circuit, and HLS for computation blocks, which are used to perform the normalization of a frame needed for the CNN. Results outperform previous implementations of frames collection and normalization using ARM processors running at 800MHz on a Zynq7100 in both latency and power consumption. A measured 67% speedup factor is presented for a Roshambo CNN real-time experiment running at 160fps peak rate.Comment: 7 page

EdgeDRNN: Enabling Low-latency Recurrent Neural Network Edge Inference

This paper presents a Gated Recurrent Unit (GRU) based recurrent neural network (RNN) accelerator called EdgeDRNN designed for portable edge computing. EdgeDRNN adopts the spiking neural network inspired delta network algorithm to exploit temporal sparsity in RNNs. It reduces off-chip memory access by a factor of up to 10x with tolerable accuracy loss. Experimental results on a 10 million parameter 2-layer GRU-RNN, with weights stored in DRAM, show that EdgeDRNN computes them in under 0.5 ms. With 2.42 W wall plug power on an entry level USB powered FPGA board, it achieves latency comparable with a 92 W Nvidia 1080 GPU. It outperforms NVIDIA Jetson Nano, Jetson TX2 and Intel Neural Compute Stick 2 in latency by 6X. For a batch size of 1, EdgeDRNN achieves a mean effective throughput of 20.2 GOp/s and a wall plug power efficiency that is over 4X higher than all other platforms.Comment: This paper has been accepted for publication at the IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Genoa, 202

LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.Comment: Submitted to ISCAS2023, 4 pages, plus references, github link provide

NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps

Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving many state-of-the-art (SOA) visual processing tasks. Even though Graphical Processing Units (GPUs) are most often used in training and deploying CNNs, their power efficiency is less than 10 GOp/s/W for single-frame runtime inference. We propose a flexible and efficient CNN accelerator architecture called NullHop that implements SOA CNNs useful for low-power and low-latency application scenarios. NullHop exploits the sparsity of neuron activations in CNNs to accelerate the computation and reduce memory requirements. The flexible architecture allows high utilization of available computing resources across kernel sizes ranging from 1x1 to 7x7. NullHop can process up to 128 input and 128 output feature maps per layer in a single pass. We implemented the proposed architecture on a Xilinx Zynq FPGA platform and present results showing how our implementation reduces external memory transfers and compute time in five different CNNs ranging from small ones up to the widely known large VGG16 and VGG19 CNNs. Post-synthesis simulations using Mentor Modelsim in a 28nm process with a clock frequency of 500 MHz show that the VGG19 network achieves over 450 GOp/s. By exploiting sparsity, NullHop achieves an efficiency of 368%, maintains over 98% utilization of the MAC units, and achieves a power efficiency of over 3TOp/s/W in a core area of 6.3mm$^2$. As further proof of NullHop's usability, we interfaced its FPGA implementation with a neuromorphic event camera for real time interactive demonstrations

Práctica de diseño hardware/software de un robot móvil con interfaces inalámbricas

En el presente artículo se describe una práctica de laboratorio multitarea en el ámbito de las asignaturas de sistemas empotrados en los grados de Ingeniería Informática, mediante una metodología de gestión de proyectos basada en Kanban. La práctica abarca diferentes familias de microcontroladores de distintos niveles de dificultad de programación, lectura de diferentes tipos de sensores con distintas interfaces, comunicación inalámbrica y control de motores. Esta práctica se enfoca como la elaboración de un proyecto en el que los alumnos han de ir realizando mediante tareas que inicialmente se planifican utilizando la metodología Kanban. En concreto, el desarrollo de la práctica se basa en la elaboración de un robot móvil controlado remotamente y de forma inalámbrica. El sistema de divide en tres partes: el dispositivo de control que cuenta con un microcontrolador tipo Arduino y dos joysticks analógicos como interfaz de usuario, el robot móvil que utiliza un microcontrolador STM32 con un RTOS (Real Time Operating System) con el que se realiza la lectura de los diferentes sensores que irán embarcados en el robot además de manejar el controlador de motores para un motor DC para la velocidad y un servo para el control de la dirección. Para la comunicación inalámbrica se utilizan módulos de radio de 2.4GHz de la familia XBee Pro Serie Z2B. Por último, se diseñará una aplicación software de escritorio bajo un sistema operativo Windows escrita en lenguaje C# utilizando .NET Framework y WPF (Windows Presentation Foundation), que mostrará la información que el robot envía de cada uno de sus sensores. El PC donde está alojada la aplicación tiene conectado un módulo XBee, anteriormente mencionado, con el que se comunica mediante una conexión serie virtual (VCP). Para implementar la metodología Kanban se hará uso de una herramienta online y gratuita llamada Trello que permite la creación de diferentes tableros en el que ir añadiendo tareas (mediante tarjetas) e irlas moviendo entre las diferentes columnas según el estado de ésta. A cada tarea se le puede añadir uno o más participantes además de ponerle una fecha de vencimiento entre otras opciones. En el desarrollo de este tipo de prácticas se añade la dificultad del manejo de diferentes entornos de desarrollo, uno por cada tipo de microcontrolador y el de la aplicación software. Esta práctica se ha dividido en varias sesiones y ha presentado un gran atractivo para el alumnado ya que se consigue un sistema funcional y muy ampliable al final de estas.This paper presents a laboratory session of embedded systems imparted in the Computer Science degree using Kanban, a project management methodology. In the laboratory session different microcontroller families are used for reading several sensor types, wireless communications and motor control. This session is focused like a project in which the students have to complete the task previously described using Kanban. The project consist on implementing a mobile robot that is handled using a wireless controller. The system is divided in three parts: the controller device that is designed using an Arduino microcontroller to read two analogical joysticks used by the user, the mobile robot that uses a STM32 microcontroller with a RTOS (Real Time Operating System) to read the sensors attached to the robot and to handle the motor controller for a DC motor to control the velocity and, finally, a servo motor to change the robot direction. Some 2.4GHz radio modules of the XBee Pro Serie Z2B are used to implement the wireless communication. Finally a C# WPF Windows application is implemented using .NET framework, which collects the information from on-board sensors. An XBee module is plugged in the computer where the application runs using a virtual communication port (VCP). To plan the project under the Kanban methodology, an online free tool called Trello is used. Trello lets the user create different panels in which cards can be added and moved between different columns that denote the state of each card. Cards allow to add several participants and a due date. In this laboratory session the students have to learn several development environments which presents an extra difficulty. The laboratory session has been divided in several practical sessions and the students have been very motivated during every of them because at the end they obtain a functional robot which can be extended with new sensors

Realización de un proyecto en grupo con carácter multidisciplinar para alumnos de Ingeniería de la Salud usando la metodología ABP

La sociedad avanza, y este avance favorece la aparición de nuevas necesidades, las cuales son cubiertas por profesionales especializados en campos específicos, como médicos, ingenieros, profesores, etc. Pero cada vez más, estos nuevos “problemas” requieren soluciones complejas y multidisciplinares, haciendo uso de varios campos de conocimiento. Es por esto que, en los últimos años, se hayan creado nuevas titulaciones en el ámbito universitario para formar a profesionales que posean los conocimientos necesarios para afrontar estos nuevos retos. Este es el caso del grado en Ingeniería de la Salud, el cual tiene un carácter multidisciplinar, combinando conocimientos biomédicos aplicados a la ingeniería informática e ingeniería en general. Los alumnos que eligen esta titulación se pueden catalogar en dos perfiles diferenciados: los procedentes de la rama científica-tecnológica y los de ciencias de la salud del bachillerato. Dado que las asignaturas de la titulación en los primeros años son de formación básica y se dividen prácticamente en 50% de una rama y 50% de la otra, según la procedencia del alumno puede que encuentre mayor dificultad en las asignaturas que no son de la suya en concreto. En el curso académico 14/15, propusimos la realización de un proyecto en grupo cuyos integrantes estaban combinados entre alumnos de la rama científica-tecnológica y de la salud, y cuyo tema requería conocimientos de ambos campos, de forma que hubiera una colaboración entre ambas partes. La finalidad de este proyecto era que los alumnos asimilaran los conceptos teóricos que eran nuevos para ellos aplicándolos a una situación real apoyándose en el resto de los miembros del grupo. El proyecto se desarrolló siguiendo la metodología de aprendizaje ABP aplicada en las sesiones prácticas de la asignatura, marcándose en cada una de estas sesiones una serie de hitos que los alumnos deberán ir completando. Se midió el grado de satisfacción de los alumnos mediante un cuestionario, obteniendo como resultado un alto grado de satisfacción.Society progresses very quickly, and that progress favors the emergence of new needs. These needs are provided, mostly, by experts in specific fields, such as doctors, engineers, teachers, etc. But increasingly, these new "problems" require complex and multidisciplinary solutions, involving different fields of knowledge. This is the main reason why, in recent years, new degrees have appeared, educating future experts in these fields to be able to face these new challenges. This is the case of Health Engineering Degree, which has multidisciplinary contents, combining biomedical knowledge applied to computer science and engineering. The students in this degree can be differentiated into two different categories, depending the high school studies selection: those from the scientifictechnological area and those from the health science area. Since the subjects of the first course of this degree are divided almost 50% from one area and 50% of the other, according to previous studies, the student may find more difficulty in subjects that are not in the their knowledge field. In the academic year 14/15, the development of a project in working groups whose members were combined between students of scientifictechnological education and students of health science was proposed. This work required knowledge of both fields, so they had to collaborate. The purpose of this project was that the students assimilated the theoretical concepts that were new to them applying the new concepts to a real situation supporting on the rest of the group. The project was development following the learning methodology PBL (Problembased learning) applied in practical sessions with milestones that students must complete. Both the satisfaction and motivation of the students were measured by a questionnaire, obtaining very good results, meaning that the students enjoyed collaborating with other teammates to face a multidisciplinary project

Práctica de desarrollo de interfaces hardware/software para la monitorización del estado de un PC

Este artículo presenta una práctica laboratorio impartida mediante una metodología de aprendizaje basado en proyectos (ABP) [1] para dotar de la capacidad de diseñar y desarrollar un monitor del estado de un ordenador, integrado en un sistema empotrado que se comunica con una aplicación de escritorio, a nuestros alumnos de la asignatura de Diseño de Microcontroladores (DM) en el contexto del Máster en Ingeniería de Computadores y Redes. Esta práctica abarca la comunicación Hardware/ Software entre un microcontrolador con un núcleo Cortex-M4 y una aplicación software escrita en lenguaje C# usando el entorno Visual Studio Community 2015 a través de puertos series virtuales (VCP). Esta práctica está enfocada como un proyecto que los alumnos han de ir realizando desde cero, avanzando mediante la consecución de hitos, hasta conseguir obtener un sistema final. El sistema a desarrollar se divide en dos partes, por un lado tenemos un PC con un sistema operativo de la familia Windows, en el que se construye una aplicación visual mediante Windows Forms, la cual obtiene información del sistema de forma periódica y la envía al microcontrolador mediante comandos usando el puerto serie (USB o comunicación Bluetooth). Por otro lado tenemos un microcontrolador de la familia STM32 que dispone de un display LCD ejecutando una plataforma completamente libre, .NET Micro Framework, la cual recibe a través del puerto serie la información obtenida gracias a la aplicación software del PC y la muestra en la pantalla, obteniendo así una herramienta de monitorización del PC sin tener que estar conectado físicamente a éste. El desarrollo de este tipo de proyectos se añade la dificultad de la necesidad del uso de diferentes herramientas para el desarrollo del firmware y del software en paralelo, de manera incremental, y enfocadas para ámbitos de uso muy distintos. Esta práctica ha tenido una gran acogida por parte de los alumnos, ya que les ha servido de ejemplo del desarrollo de firmware para un microcontrolador usando la plataforma .NET MF y de su comunicación con el PC por medio de una aplicación visual.This manuscript presents a practical laboratory session imparted using a project-based learning methodology (PBL) to provide the capacity of designing and developing a computer status monitoring device, integrated in an embedded system that communicates with a desktop software tool, to our students in the Computer Engineering Master’s Degree. This practice session encompasses Hardware/ Software communication between a microcontroller with a Cortex-M4 kernel and a desktop software application through virtual COM ports (VCP) written in C# using Visual Studio Community 2015. This lab session is focused as a project that students must be making from scratch by achieving and completing some milestones to obtain a final functional system. The project is divided into two different parts. First, we have a Windows PC where a visual software application that gathers information from the system and sends it periodically to the microcontroller (USB or Bluetooth) has to be built using Windows Forms. On the other hand, we have a microcontroller from the STM32 family that has a 2.4’ LCD display executing .NET Micro Framework that receives the information obtained from the PC through the serial port and displays it in the screen. This way, students create a computer status monitoring tool that does not need to be connected physically to it to receive the information. The development of this project is added to the need of using different tools for firmware and software development, focused to very different fields of use. This practice has been well received by the students, because it has served as an example of the firmware development for a microcontroller using the .NET MF platform as well as the communication between the PC and the microcontroller using a visual software application

Incremento de la motivación en el aprendizaje de enseñanzas informáticas mediante herramientas maker

La enseñanza de asignaturas del ámbito de la informática está experimentando un gran cambio. La motivación del alumnado se encuentra en proceso de una recesión. Ésta, históricamente, se ha relacionado con el alumnado, pero no es del todo correcto. Es obligación del profesor aportar argumentos en la docencia para facilitar esta motivación. En este trabajo se ahonda acerca de esta discusión, incidiendo en las posibles causas. Como caso práctico, se explica la metodología implementada en una asignatura impartida en dos centros (y ciudades) diferentes y con idéntico contenido. Esta asignatura pertenece a un grado compartido entre Universidad de Sevilla y otra universidad, impartiéndose en ambas sedes por igual y con un temario idéntico. Mientras ésta asignatura posee históricamente un enfoque más clásico, desde la Universidad de Sevilla se ha dado un giro en su metodología de docencia. Esta actualización de contenidos y aprendizaje basado en proyectos se detalla aquí, así como los resultados obtenidos.The teaching of subjects in the informatics field is undergoing a drastic change. Student motivation is experimenting a recession. Motivation historically has been associated to students, but that statement is not entirely correct. The obligation of providing arguments in order to motivate students is a part of the teachers work too. This paper delves into this discussion, focusing on the possible causes of these problems. As a practical matter, it delves into the teaching methodology implemented in a subject taught in two different centers (and cities), but with the same content. This subject is imparted in an engineering degree that is part of a joint project between the University of Seville and other one, so it is imparted in both sites equally and with an identical agenda. While this subject has a more classical approach at most universities, from the University of Seville its teachers have modified the teaching methodology. This content update, the project-based learning and the obtained results are detailed here